1. Field of the Invention
The present invention mainly relates to an image processing apparatus and an image processing method used for an electronic image pickup device, such as a digital camera. In particular, the present invention relates to an image processing apparatus, an image processing method, and a distortion correcting method, in which a distortion correcting function is realized without increasing the circuit scale and the amount of transfer data and the spatial position of a small region can be calculated in the distortion correction processing in units of a small region (e.g., block line).
2. Description of the Related Art
Irrespective of a digital camera or a silver-halide film camera, the distortion aberration generally appears in a camera optical system. Further, most types of cameras currently-released have an optical zoom function. In such a case, the distortion aberration state changes from the wide end to the tele end. At the wide end, a barrel distortion frequently appears. At the tele end, a pincushion distortion frequently appears.
For example, a grid subject is photographed and then the distortion aberration is observed as the barrel distortion or pincushion distortion.
FIG. 56 shows a grid subject. FIG. 57 shows a photographed image with the barrel distortion. FIG. 58 shows a photographed image with the pincushion distortion.
Meanwhile, in the digital camera, data of an image pickup element, such as a CCD, is subjected to various image processing, then, the data is compressed in a JPEG compression format, and the compressed data is recorded to a recording medium, such as a memory card.
FIG. 59 shows the concept of image processing sequence in the general digital camera. An image pickup signal captured by the CCD is subjected to pixel defect processing and A/D conversion in the pre-processing. The obtained image data is temporarily stored in a frame memory, such as an SDRAM. Subsequently, the image data read from the frame memory is subjected to various image processing. Further, the image data is compressed in JPEG processing and is recorded in a memory card, serving as a recording medium.
FIG. 60 is a block diagram showing a conventional image processing apparatus for digital camera (according to e.g., one conventional art disclosed in Japanese Unexamined Patent Application Publication No. 2000-312327).
Referring to FIG. 60, the conventional image processing apparatus has a bus 103, to which a CPU 104, a pre-processing circuit 102, a plurality of image processing circuits 106-1 to 106-n, a JPEG processing unit 107, a frame memory 105 and a memory card 108, serving as a recording medium, are connected. Then, under the control of the CPU 104, an image pickup signal from a CCD 101 is subjected to pixel defect processing and A/D conversion by the pre-processing circuit 102 and is then temporarily stored in the frame memory 105 via the bus 103. Subsequently, the image data is read from the frame memory 105, and the data is inputted to the image processing circuit 106-1 via the bus 103 to be subjected to predetermined image processing. Then, the data is rewritten again to the frame memory 105 via the bus 103. Similarly, the data is received and transmitted between the frame memory 105 and the image processing circuits 106-2 to 106-n via the bus 103. Finally, the JPEG processing unit 107 JPEG-compresses the data, and the compressed data is temporarily stored in the frame memory 105. The processed data read from the frame memory 105 is recorded in a memory card 108. In each image processing and JPEG processing, the image processing is performed in units of small region (block line).
In at least one of the image processing circuits, with the distortion correcting method as described with reference to FIGS. 56 to 58, the following distortion correcting formula [Formula 5] optically expressed is used, thereby correcting the distortion aberration of the picked-up image.
                                          Z            2                    =                                                    (                                  X                  -                                      X                    d                                                  )                            2                        +                                          (                                  Y                  -                                      Y                    d                                                  )                            2                                      ⁢                                  ⁢                  {                                                                                          X                    ′                                    =                                                                                    (                                                  X                          -                                                      X                            d                                                                          )                                            ·                                              [                                                  1                          +                                                      A                            ·                                                          Z                              2                                                                                                      ]                                                              +                                          X                      d                                                                                                                                                                Y                    ′                                    =                                                                                    (                                                  Y                          -                                                      Y                            d                                                                          )                                            ·                                              [                                                  1                          +                                                      A                            ·                                                          Z                              2                                                                                                      ]                                                              +                                          Y                      d                                                                                                                              [                  Formula          ⁢                                          ⁢          5                ]                            (X, Y): Coordinates in corrected image        (X′, Y′): Coordinates in picked-up image        (Xd, Yd): Distortion center        A: Distortion correcting coefficient        
However, in the distortion correction using [Formula 5] optically expressed, the corrected image has a data unused/data shortage portion.
FIGS. 61 and 62 explain the operation for correcting the distortion aberration of the picked-up images having the barrel distortion and the pincushion distortion with [Formula 5].
Referring to FIG. 61, an original image (on the left in the drawing) with the barrel distortion shown by a broken line is corrected by [Formula 5], and the corrected image (on the right in the drawing) is having an unused portion out of an image output range.
Referring to FIG. 62, an original image (on the left in the drawing) with the pincushion distortion shown by a broken line is corrected by [Formula 5], and the corrected image (on the right in the drawing) is smaller than the image output range and causes a data shortage portion.
Against the inconvenience shown in FIGS. 61 and 62, Japanese Unexamined Patent Application Publication No. 9-098340, serving as a prior application, discloses such a technology that the decompressed portion, as a result of the correction, is reduced by the electronic zoom operation after the distortion correction processing and is returned to the original position, thereby suppressing image information lost as a result of distortion correction processing at the minimum level.
Generally, the reading operation of an image pickup element, such as a CCD, includes two types of reading operation: (1) reading operation of all pixels for main image pickup; and (2) thinning-out and reading operation of through image. Since the image data which is partly stored from the picked-up data by the thinning-out and reading operation does not have (1:1) aspect ratio, the distortion correction processing of the image data without additional processing is not precisely performed.
Further, another conventional technology for distortion correction, as a part of the image processing, is disclosed in, e.g., Japanese Unexamined Patent Application Publication No. 6-181530, and Japanese Unexamined Patent Application Publication No. 10-224695.
As disclosed in Japanese Unexamined Patent Application Publication No. 6-181530, when detecting means detects that the image pickup position of an image pickup zoom lens in the image pickup operation is within the position with a large distortion aberration, the geometric distortion of the image captured through the image pickup zoom lens is corrected by reading the image pickup data of a solid-state image pickup element based on geometric modification.
Further, as disclosed in Japanese Unexamined Patent Application Publication No. 10-224695, a solid-state image pickup device receives subject light via an optical system and picks up an image of the subject. The image-picked-up data is stored in a random access video memory, a random-reading timing generating circuit, having distortion correction data for correcting the aberration generated by the optical system, reads a signal generated by the solid-state image pickup device in a predetermined order based on the distortion correction data, and generates a video signal, thereby correcting the distortion aberration generated by the optical system.